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ABSTRACT: Directed energy deposition (DED) is a major metal additive manufacturing (AM) technology that is 
increasingly used in many industries due to its ability to manufacture complex components of arbitrary shapes and 
sizes. However, a lack of timely geometry assessment and the consequent geometry control hinders the 
development of DED towards zero defect manufacturing. In this study, a real-time geometry assessment 
methodology is developed for laser pow-der directed energy deposition (LP-DED). A geometry assessment system 
is developed using a laser line scanner capable of inspecting the melt pool area, the just solidified area, as well as 
layer-wise inspection. An image processing method with an encoder-decoder based profile completion network 
was developed to obtain accurate track profile in images from real-time inspection. Experiments have been 
conducted to validate the proposed methodology by depositing multi-layer X-shape objects. 
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1. INTRODUCTION 


Metal additive manufacturing (AM) technologies’ potential to revolutionize the manufacturing industry has not 
only been well-recognized but has inspired many fields [1]. According to the American Society for Testing and 
Materials (ASTM) standard [2], the two main groups of met-al additive manufacturing technologies are directed 
energy deposition (DED) and powder bed fusion (PBF), both of which have been widely applied in, for example, 
the aerospace, automobile, and biomedical industries. In recent years, DED has gained attention as a viable 
manufacturing method in the construction industry where metallic materials are used extensively in distinctive and 
complex designs [3]. Often, traditional techniques such as hot rolling, cold forming, and extrusion can only 
produce regularly shaped, prismatic metallic components [4], which limits the potential use of metallic materials in 
construction and design. DED can complement traditional methods and produce components with almost any 
shapes with high precision. 


However, DED is somewhat plagued by geometry problems [5]. For example, the thicknesses of the deposited 
(“printed”) layers often deviate from their design values [6]. When more and more layers are deposited, heat 
accumulation tends to cause the layers to spread, increasing their width but decreasing their height [7]. In addition, 
as the nozzle comes to deposit the object’s comers or intersections, the resulting geometry also quite often deviates 
from the design [8]. The geometrical dimensions of the deposited object often fail to meet the quality requirement 
or even collapse when geometry deviation occurs during the deposition and is not solved in a timely manner, which 
will lead to time and cost waste. Moreover, an accurate geometry profile is quite helpful for other quality analysis 
during the printing process, such as the online stress measurement [9]. Therefore, it is important that the geometry 
of the printed object is continuously inspected in real time, as more material gets deposited. 


Vision cameras and laser line scanners are often employed to assess the geometry of an object being printed via 
DED [10]. However, though vision cameras can assess the geometry in real-time, only either the track width or 
track height is measured, but not the whole track pro-file [11]. On the other hand, laser line scanners are mostly 
used post-DED, at which point the influence of powder and deposition laser is quite small so that the measuring 
accuracy is high. Besides, previous studies on geometry assessment have tended to focus on single- or multi-layer 
straight line deposition; multi-layer deposition of components with sharp features such as intersections or corners, 
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is important but not well studied. Therefore, the objective of this study is to develop a system capable of 
conducting real-time 3D geometry assessment during multi-layer deposition of objects with sharp features. There 
are four main contributions: (1) A geometry assessment system was developed to achieve both real-time inspection 
and layer-wise inspection during the LP-DED process; (2) In real-time inspection, an image processing method 
including a novel encoder-decoder based profile completion network was developed to obtain an accurate track 
profile; and (3) geometry assessment of multi-layer X-shape deposition has been achieved. 


This paper is organized as follows. Section 2 gives some background on our research, which includes the working 
principle of a laser line scanner, a description of LP-DED and geometry assessment during DED. Section 3 
explains the developed geometry assessment methodology. Section 4 provides the experimental results and 
discussion of the proposed methodology. Section 5 concludes the paper by presenting a summary, limitations, and 
future work. 


2. RESEARCH BACKGROUND 
2.1 Description of laser line scanner 


A laser line scanner is a piece of non-destructive testing (NDT) equipment that has been used successfully to 
capture the shape of an object. The scanner operates based on the principle of laser triangulation as shown in 
Figure 2-2. By selecting the bottom of the sensor as a reference as shown on Figure 2-2, the basic triangulation 
calculation is expressed in equation (1). 


ay 


(1) 


b, = 
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For laser line scanner, instead of projecting a single point, a laser line is projected on the target surface. After that, 
the diffusely reflected light of the laser line is detected by a high-quality sensors array called CMOS sensor matrix. 
Each projected point corresponds to one column on the sensor matrix. Based on the position of the detected laser 
beam on the corresponding column on sensor matrix, the distance of one measuring point to a defined reference in 
the sensor (Z coordinate) and can be calculated via triangulation, and the exact position of each point on the laser 
line (X coordinate) is acquired accordingly. 


Light source 


Z-axis CMOS array 


Receiver 


(a) (b) 


Fig. 1: (a) Principle of optical triangulation, (b) Principle of laser line scanner 


In addition, a band filter is embedded right before the CMOS sensor to avoid the reflection of light beyond the 
expected wavelength, and only captures the reflection of the projected laser line. With respect to laser light, red and 
blue laser diodes are commonly available for laser line scanners. The red laser scanner is ideal for common 
measurement tasks especially with extremely dark surfaces whereas the blue laser scanner is ideal for transparent, 
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organic, and red-hot glowing surfaces. In the DED process, when the high power-density laser is focused on a 
continuous stream of metal powder, the substrate becomes red- hot glowing surface. Thus, for this study a blue 
laser diode line scanner is used to achieve an accurate result. In addition, the blue laser is preferred to the red one 
since it is an extremely sharp-focused laser line that does not penetrate the surface. 


2.2 Geometry assessment during DED 


The geometry assessment targets include the geometry of the melt pool area and the just solidified area as well as 
the layer geometry of the deposited object. Much research in recent years has focused on inspecting the 
geometry of the melt pool, since it can be used for real-time geometry control [10]. Camera-based methods are 
popularly used to inspect the melt pool geometry, including vision camera and infrared camera. Vision cameras 
can be used to measure either the width or the height of the melt pool depending on the installation location and 
the target measurement field [11], while infrared cameras are normally used to measure the width and length of 
the melt pool [17]. Some recent papers have attempted to estimate the melt pool height using infrared cameras 
with the help of deep learning methods. However, these methods cannot measure the spatial profile of the melt 
pool area. The inspection of just solidified area is important as it is reported that this can be used for online stress 
estimation [9]. Moreover, compared with the melt pool area, the just solidified area might better represent the 
final geometry of the deposited track since thermal shrinking occurs during the solidification process [18]. The 
just solidified area is quite close to the melt pool area so that some studies use it for real-time geometry control 
though there is bound to be a small lag. Similarly, camera-based methods are used but a spatial profile cannot be 
obtained [19]. While laser line scanners can be used to obtain a special profile [20], its performance are affected 
by powder reflection and high intensity melting laser during LP-DED process, which prevents it from obtaining 
accurate spatial profile, thus it is often used for layer-wise geometry inspection rather than real-time geometry 
inspection. The inspection of layer geometry after printing each layer has been studied as well. This method 
needs to consider inspection path during the design process, which increases design effort and printing time. 
However, it is applicable to all kinds of printing shapes and toolpaths. In addition, for layer-wise control which 
does not need instant feedback, it is a better choice [21]. To address geometry assessment of all three targets, a 
geometry assessment system is developed in this study which aims to achieve both real-time inspection and 
layer-wise inspection of the track profile using laser line scanner. 


Note that the term “real-time” might be ambiguous, as it has been used in a different sense in the literature from 
field to field. In this study, inspections with the laser line of a line scanner located at just a solidified area or 
melting pool area are called real-time inspection, where a small delay relative to the deposition is expected. On 
the other hand, inspection that takes place after each layer has been printed is called layer-wise inspection, since 
normally a large delay exists. 


3. METHODOLOGY 
3.1 Overview of the geometry assessment methodology 


For real-time inspection, images are captured during deposition. Here there are three apparent problems that 
would prevent us from obtaining an accurate profile: (1) powder reflection, (2) melt-influenced area and (3) track 
profile missing, as illustrated in Fig. 4. The following steps are proposed to overcome these three problems. 
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Fig. 4 Problems with raw images: (a) typical profile from real-time inspection of melt pool area, (b) typical 
profile from real-time inspection of just solidified area 


3.2 Powder reflection removal 


Firstly, an image enhancement technique, namely contrast stretching, is applied to improve the contrast in each 
image by converting the original intensity value of a smaller range to a larger range of intensity values, as shown 
in Eq. (3) and Fig. 5. 


s=T(r) (3) 


where r is the input intensity, s is the output intensity, and T is the intensity transformation function. By applying 
this function, the brighter pixels become even brighter, and darker pixels become even dimmer. Since powder 
reflection always has lower intensity, it can remove the pixels of powder reflection. 


Output intensities, s 


Input intensities, r 


Fig. 5 Powder reflection removal: contrast stretching 


3.3 Melt-influenced area removal 


To remove the melt-influenced area, a DBSCAN clustering algorithm is applied on the image. As shown in Fig. 
6(a), for all non-zero pixels in the image, a core pixel is selected if the pixel has n number of neighbors, where 
the neighbors are pixels within a distance ¢ from the core pixel. The distance between two pixels is calculated 
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using Eq. (4): 


dy = [y -u + -oY a) 


where (u; v;) and (uj, vj) are the U-V coordinates of pixels i and j. A cluster is formed by recursively 
taking a core pixel, finding among all of its neighbor pixels that are core pixels, in turn finding all of their 
neighbors that are core pixels, and so on. After clustering, the cluster with the largest mean v value will be 
removed. For some images, the melt-influenced area will be connected to the track profile, and the track profile 
will be removed as shown in Fig. 6 (b). To solve this problem, a modified DBSCAN is proposed by adding an 
additional coordinate m to represent the distance to the lowest non-zero pixels for each column as illustrated in 
Fig. 6 (c) and the distance between two pixels is calculated using Eq. (5). After that, the DBSCAN algorithm is 
applied, and the result can be seen in Fig. 6(d) 


(5) 


Neighbors 


Core pixel 


(a) 


Define cdordinate m 


(c) 


Fig. 6 Melt-influenced area removal: (a) explain DBSCAN clustering on the image, (b) directly apply DSCAN, 
(c) add coordinate m and (d) apply modified DBSCAN 


3.4 Encoder-decoder based profile completion 


After melt-influenced area removal, the track profile is extracted. However, there exists severe profile missing 
problems, which will severely deteriorate the quality of 3D point cloud for geometry assessment. The profile 
missing problem is mainly due to bad reflection on the deposited metal surfaces, especially when there is 
high-intensity laser radiation from the printer. As shown in Fig. 7, the cross-sections at different locations on the 
X-shape represents different types of profile missing, and a pattern can be observed for each track: (1) at the 
beginning of each track (location A), the opposite side of the melt-pool influenced area are missing for the 
one-peak cross section; (2) when approaching the intersection (location B), the line scanner captures two-peak 
cross section and the middle part of the two-peak cross section is missing; (3) at the intersection (location C), the 
top part of the one-peak cross section is missing; (4) when getting away from the intersection (location D), two 
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side parts of the two-peak cross section are missing; (5) at the end of each track (location E), same as in location 
A, the opposite side missing of one-peak cross-section can be observed. For such varied types of profile missing 
problems, traditional completion methods such as curve fitting [22] or interpolation methods cannot achieve 
satisfied result, especially for the middle part missing of two-peak cross section. 


Fig. 7 Track profile missing problem 


To complete the track profile in captured images, an encoder-decoder based profile completion algorithm is 
proposed. The idea of adopting an encoder-decoder based network comes from point cloud completion techniques. 
Encoder-decoder based networks are widely used in 3D point cloud completion tasks [23], [24], as the encoder can 
summarize the geometric information from an incomplete input point cloud to form a feature vector, and, based on 
the feature vector, the decoder will predict the complete shape of the point cloud. The proposed algorithm is 
revised from DeepLabv3+ [25], which is a popular encoder-decoder network with images as input. 


| Encoder 
Concatenation 


1280x33x33 


Multiscale Dilated 
Convolution 256x33x33 


Concatenation 


304x129x129 


2048x33x33 


1024x33x33 
512x65x65 
ResNet101 256x129x129 


Menn. eee 


256x129x1 


9 


1x513x513 


a Normal Convolution + Batch Normalization + ReLU a Dilated Convolution + Batch Normalization + ReLU 


d 1x1 Convolution + Batch Normalization + ReLU oy Pooling SZ Upsampling 


Fig. 8 Proposed encoder-decoder based profile completion network 
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In this study, the useful information in preprocessed images is the track profile pixels, which take up a small 
proportion of the whole image. Therefore, to make an efficient use of information in the images, the images from 
previous steps are further converted to 1D array and fed as input. A novel deeplab1D network is proposed which 
changes 2D operation in all layers of the DeepLabv3+ network to a 1D operation. Moreover, since previous 
profiles might provide useful information for the completion of current profile, therefore the previous (¢ — 1) 
profiles are considered as input for the proposed network. When ¢ equals 1, only the current profile is considered. 


The architecture of the proposed encoder-decoder based network is shown in Fig. 8. First, the image 
of 140 x 220 pixels from previous steps is converted into a 1 x 220 array. For each column of the array of the 
image, the gravity of the intensity plot on the v axis is taken as one of the values in the 1 x 220 array. The 1 x 
220 array is inputted into the proposed network, which includes an encoder and a decoder. The encoder consists 
of a ResNetID and an ASPP1D module. The ResNet!D module is based on ResNet-101 [26]. The ASPP1D 
module has evolved from Atrous Spatial Pyramid Pooling (ASPP) [27], which conducts several parallel Atrous 
convolutions with different rates. An Atrous convolution is described in Eq. (6). 


yli] =) xli +r- k]w[k] 6) 


k 


where I is the location on the output feature map Y, W is a convolution filter applied over the input feature 
map x. The Atrous rate Y determines the stride that samples the input signal. Note that an Atrous convolution 
with r = 1 is equal to a standard convolution. In the encoder module, the concatenated output from ASPP1D 
is processed by a 1 x 1 convolution and an interpolation by a factor of 4, then concatenated with the convolved 
low-level feature from the ResNet1D. After this concatenation, a few convolutions are applied to refine the 
features followed by interpolation by a factor of 4 to recover the final output shape (a 1 x 220 array). 


3.5 Point cloud generation 


The captured data from real-time inspection and layer-wise inspection is in its local coordinate system. For 
subsequent geometry assessment, such as comparing as-designed geometry with as-built geometry, the captured 
data needs to be converted to the global coordinate system. For real-time inspection, the U-V to X-Z coordinates 
conversion and X-Z coordinate transformation are needed to obtain the point cloud of target cross-section from 
captured images. While for layer-wise inspection, 2D points are collected, thus only X-Z coordinate 
transformation is conducted. 


Since the commercial line scanner normally uses a CCD sensor to capture its emitted laser reflection on the target 
surface, this process can be modeled using a pin-hole model, which is the basis camera model based on the 
perspective projection principle [28]. 


First, intrinsic calibration is conducted, which is to reconstruct the X-Z coordinates in the real-world coordinate 
system, given the U-V coordinates in the image coordinate system using the following equation. 


x] kı” kı? key3"] pu 
a = [koi kaz” kzz’ >| (10) 
1 kzı" k32" kg3'} +1 


By getting several pairs of U-V and X-Z coordinates, a least-square solution can be used, and the intrinsic 
transformation matrix K can be obtained. 


Second, after obtaining the X-Z coordinates in the line scanner’s local coordinate system (X,,Z,), a translation 
matrix is needed to convert the X-Z coordinates into the global coordinate system (Xg, Zg), which is the extrinsic 
calibration process. After installing the laser line scanner, a rectangular calibration bar with known dimension is 
put at the origin of the global coordinate system, thus the as-designed cross section profile of the calibration bar 
in the global coordinate system can be obtained. Finally, after obtaining coordinates of measure profile in 
nozzle’s coordinate system, the 3D coordinates of the measured profile in the global coordinate system can be 
calculated by fusing the nozzle’s position using Eq. (16). 
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Xj] 1 0 tx, > 
= n 
iM -|o eaa (16) 


where the tx, ty, and tz, are the coordinates of the printing position in the global coordinate system, i.e., the 
nozzle’s position. To obtain the nozzle’s position for each captured profile, a log file is extracted from the DED 
printer during deposition which contains the location of the nozzle and a timestamp at each location. 


4. EXPERIMENT AND DISCUSSION 
4.1 Experiment setup 


The LP-DED printer used in this study is an InssTek MX-400, which is a commercial metal printer equipped 
with a 5-degree-of-freedom mechanical moving stage, a Ytterbium fiber laser with a wavelength of 1070 nm, a 
maximum power of 1kW, and a focal laser beam diameter of 800 um, as well as a metal powder delivery system 
with shield gas and carrier gas (Argon gas). A 316 L stainless-steel powder with an average particle size of 100 
um was used for deposition and a substrate using the same material as the powder with dimensions 100 mm x 50 
mm x 10 mm was placed on the moving stage. 


Two laser line scanners (Micro-Epsilon scanCONTROL 3000-25/BL), an inclined line scanner and an upright 
line scanner (Fig. 13), are installed for real-time and layer-wise inspection, respectively. Calibrations are 
conducted for both line scanners to get the transformation matrix used for 3D point cloud generation before 
deposition. For the inclined line scanner, intrinsic and extrinsic calibration are conducted; for the upright line 
scanner, only extrinsic calibration is needed. The exposure time was set to 20 ms (determined based on 
experience) to get a better reflection from the metal surfaces. The deposited object is a 20-layer X-shaped object. 
Two experiments are conducted: Experiment 1: deposition with real-time inspection of melt pool area (d = 
Omm) and layer-wise inspection; Experiment 2: deposition with real-time inspection of just solidified area (d = 
3mm) and layer-wise inspection. 


Fig. 13 Experiment setup: Geometry assessment system 


972 


4.2 Processing result 


During the deposition of each layer, the inclined line scanner with laser line pointing at the melt pool or just 
solidified area captures and transfers images to the software for processing. After the deposition of each layer, 
layer-wise inspection is conducted by the upright line scanner, and point profiles are collected. For layer-wise 
inspection, 2D points of each cross-section are obtained and 3D point cloud data can be generated using extrinsic 
calibration matrix and fusing printer information. For real-time inspection, powder reflection removal and 
melt-influenced area removal are conducted for each image. Then the developed encoder-decoder based profile 
completion network is used to complete the profile on each image and the U-V coordinates are obtained. Finally, a 
3D point cloud data of the deposition X-shape object in the global coordinate system can be generated using the 
proposed method. 


Two experiments are conducted and each with 20 layers of data collected. Experiment 1 collects 1167 images and 
Experiment 2 collects 1157 images in total. Given the traverse speed of the printer (10 mm/s) and the total tool path 
length of each deposition (993.86 mm), the actual profile rate of the real-time inspection is about 12 frames per 
second. In this study, the paired image data from experiment 1 is used for model training and validation, which is 
divided into training and validation data in a ratio of 0.8, 0.2, respectively. The collected data from experiment 2 is 
used for model testing. The training process was conducted on GPU (NVIDIA GeForce GTX1080) using Python 
3.9, Pytorch 11.7 and CUDA 11.8. Since previous profile is considered, t equals 1 to 5 are considered. For each t 
value, 80 epochs are trained, the validation dataset is used to choose the best model, then the best model is used for 
testing. 


Root mean squared error (RMSE) was calculated between the output prediction array and the ground truth array. 
The average RMSE of all profiles in each layer was obtained for each ¢ for comparison (Table 4). For higher layers, 
the RMSE becomes larger. This is due to the fact that when the layer height becomes higher, the sides of the 
deposited object will have a more inclined angle of 90 degrees, resulting in a worse reflection of the laser from the 
line scanner projected on the side of the deposited object. Thus, there will be a more serious profile missing 
problem for the higher layers, making it more difficult to complete the profile. In addition, when the absolute 
deposition height increases, the network prediction error also tends to increase, but the proportion of the error with 
respect to absolute deposition height may remain the same. Therefore, an RMSE-H ratio is calculated for each 
layer, which is the RMSE error divided by the design height (Table 4). As seen, there is not much difference in the 
RMSE-H ratio for each layer. The RMSE-H ratio of the first few layers is larger. Considering that the profiles of 
the first five layers are not seriously missing and the completion model may not be effective in such cases, the 
profile completion can be applied to the higher layers without the first five layers. When more previous layers are 
involved, there is little difference in the performance of the proposed model. A slight decrease of RMSE can be 
observed when ¢ equals 2. Therefore, the model with ¢ = 2 is finally selected as the profile completion model in our 
geometry assessment system, which can generate point cloud with an RMSE of 0.21 mm compared with the 
ground truth. 


Table 4: Test results of the proposed model for different values of t 


Test result As-designed RMSE (mm) RMSE-H ratio 
height (mm) 

1 |f=2 |t=3 |t=4 |t=5 [t=1 [t=2 Jt=3 | t=4 |t=5 
1 0.2 0.03 | 0.03 | 0.03 | 0.04 | 0.03 | 16% | 14% | 16% | 18% | 15% 
2 0.4 0.06 | 0.05 | 0.06 | 0.06 | 0.06 | 14% | 13% | 14% | 14% | 14% 
3 0.6 0.07 | 0.07 | 0.08 | 0.08 | 0.08 | 12% | 11% | 13% | 13% | 14% 

Layer 

4 0.8 0.10 | 0.09 | 0.10 | 0.10 | 0.11 | 12% | 11% | 13% | 12% | 14% 
5 1 0.11 | 0.10 | 0.11 | 0.12 | 0.13 | 11% | 10% | 11% | 12% | 13% 
6 1.2 0.13 | 0.12 | 0.14 | 0.14 | 0.16 | 11% | 10% | 12% | 11% | 13% 
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7 1.4 0.16 | 0.15 | 0.18 | 0.18 | 0.19 | 12% | 11% | 13% | 13% | 13% 
8 1.6 0.16 | 0.17 | 0.18 | 0.19 | 0.20 | 10% | 10% | 11% | 12% | 13% 
9 1.8 0.20 | 0.18 | 0.20 | 0.19 | 0.21 | 11% | 10% | 11% | 11% | 12% 
10 2 0.19 | 0.18 | 0.20 | 0.19 | 0.21 | 9% 9% | 10% | 10% | 11% 
11 2.2 0.22 | 0.20 | 0.22 | 0.22 | 0.23 | 10% | 9% | 10% | 10% | 11% 
12 2.4 0.23 | 0.23 | 0.23 | 0.22 | 0.25 | 10% | 10% | 10% | 9% | 10% 
13 2.6 0.25 | 0.24 | 0.25 | 0.23 | 0.26 | 9% 9% 9% 9% | 10% 
14 2.8 0.28 | 0.29 | 0.28 | 0.30 | 0.31 | 10% | 10% | 10% | 11% | 11% 
15 3 0.27 | 0.27 | 0.28 | 0.30 | 0.33 | 9% 9% 9% | 10% | 11% 
16 3.2 0.31 | 0.29 | 0.30 | 0.31 | 0.33 | 10% | 9% 9% | 10% | 10% 
17 3.4 0.38 | 0.36 | 0.36 | 0.38 | 0.37 | 11% | 10% | 11% | 11% | 11% 
18 3.6 0.40 | 0.38 | 0.39 | 0.39 | 0.39 | 11% | 11% | 11% | 11% | 11% 
19 3.8 0.40 | 0.38 | 0.38 | 0.41 | 0.41 | 11% | 10% | 10% | 11% | 11% 
20 4 0.43 | 0.42 | 0.42 | 0.44 | 0.45 | 11% | 11% | 10% | 11% | 11% 
Overall 0.22 | 0.21 | 0.22 | 0.22 | 0.24 


5. CONCLUSION 


This study has developed a real-time geometry assessment methodology for LP-DED using laser line scanner. A 
geometry assessment system has also been developed to achieve real-time inspection of the melt pool area and the 
just solidified area, as well as layer-wise inspection of the layers’ geometry. An image processing method has been 
proposed including powder reflection removal, melt-influenced area removal and an encoder-decoder based 
profile completion network to obtain track profile on images. Then a point cloud generation method was developed 
including U-V to X-Z coordinates conversion, X-Z coordinate transformation, and 3D point cloud generation by 
fusing printer information. Experiments have been conducted to validate the proposed method and the result shows 
that an average RMSE of 0.21 mm can be achieved from point cloud comparison between the point clouds 
obtained in realtime and obtained layer-wise. The proposed real-time inspection method was able to achieve better 
performance compared with the line scanner’s built-in method, and the developed encoder-decoder profile 
completion model has been validated which outperforms baseline model. The deposition heights of the melt pool 
and solidified layer were compared, and the result showed that the differences were not significant using the 
proposed method. 
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